Metrics now have an optional minimal_effect argument that is used to compute the sample size required to reach 80% power.

REST API example:

Python API example:

from epstats.toolkit import Experiment, metric

experiment = Experiment(
    id="test-conversion-with-minimum-effect",
    control_variant="a",
    metrics=[
        Metric(
            id=1,
            name="Clicks per User",
            nominator="count(test_unit_type.unit.click)",
            denominator="count(test_unit_type.global.exposure)",
            minimum_effect=0.1,
        ),
        Metric(
            id=2,
            name="Purchases per user",
            nominator="count(test_unit_type.unit.purchase)",
            denominator="count(test_unit_type.global.exposure)",
            minimum_effect=0.1,
        ),
    ],
    checks=[],
    unit_type=test_unit_type,
)

Btw in the last PR #40, we talked about Bonferroni vs Holm-Bonferroni correction. Holm-Bonferroni can be applied here because we already have the $p$-values. However, it would result in each variant having very different required_sample_size because the correction depends on the $p$-value. I think it's better to just stick with the classic Bonferroni and use the most conservative $\alpha$ for all variants so that the required sizes are equal.

Consider an example with 4 variants and $p$-values $p_B = 0.001, p_C = 0.005, p_D = 0.01$.